Upstream tracking#165
Draft
grahamc wants to merge 3101 commits into
Draft
Conversation
ci: update DSP on releases
(cherry picked from commit 9205295)
…tenance [Backport 2.34-maintenance] installer: Bail out early if running on macOS < 14.0
Based on upstream 0b29401.
Add some calls to addTempRoot()
Flake lock file updates:
• Updated input 'nixpkgs':
'https://api.flakehub.com/f/pinned/NixOS/nixpkgs/0.2511.909248%2Brev-0590cd39f728e129122770c029970378a79d076a/019ce32b-8ace-7339-b129-cceaa8dd10c6/source.tar.gz' (2026-03-11)
→ 'https://api.flakehub.com/f/pinned/NixOS/nixpkgs/0.2511.914302%2Brev-1d4e0f865d68258aada31e68e6d79c8c463f3b34/019f1c78-b5ab-7af2-8516-c0d5406b0646/source.tar.gz' (2026-06-29)
Since curl/curl@2a2104f libcurl now swallows events in curl_multi_perform, so there's a chance that we miss a wakeup. This is somewhat reproducible on my machine by doing nix flake prefetch https://channels.nixos.org/nixos-25.11/nixexprs.tar.xz. Another option is to bring back our own wakeup pipe, but that's less portable. Ideally this regression would be fixed in libcurl too... (cherry picked from commit 0d0c333)
…tenance [Backport 2.34-maintenance] libstore: Fix libcurl thread wakeup with curl >= 8.21
flake.lock: Update
Nixpkgs now uses structured attrs (i.e. env.<VAR>), so this override wasn't working anymore. It makes a huge performance impact once GC kicks in (e.g. it speeds up a 12-core `nix search nixpkgs` from 10.5 to 6.5 seconds). Assisted-by: Claude Fable 5 <noreply@anthropic.com>
…k-size Fix INITIAL_MARK_STACK_SIZE override in boehm-gc
nix search: Print total number of derivations
The previous implementation used a std::map with traceable_allocator, which costs one GC_MALLOC_UNCOLLECTABLE() and one GC_FREE() per distinct attribute name. Uncollectable allocations always take the global GC allocation lock, making this map a major source of mutex contention during parallel evaluation (~12% of all futex calls in a 16-core 'nix search' run, plus the corresponding GC_free traffic). Instead, collect the (name, value) pairs into a plain std::vector and group them by stable-sorting on the name, which yields the same attribute order and per-attribute value order as the map. The vector doesn't need to be visible to the GC since the values it points to are kept alive by the attrsets in args[1] for the duration of the call. At 16 eval cores (with GC disabled via GC_INITIAL_HEAP_SIZE=40G), this cuts kernel time from ~21s to ~8-9s, context switches from 2.6M to 1.5M, and elapsed time from ~6.2s to ~4.9s for 'nix search nixpkgs --no-eval-cache fizzbuzz'. Single-core performance is unchanged. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
Make zipAttrsWith more parallel-eval friendly
Apply the same optimization to builtins.groupBy as to builtins.zipAttrsWith (162a34378c): instead of accumulating the groups in a std::map<Symbol, ValueVector> with traceable_allocator - which costs a GC_MALLOC_UNCOLLECTABLE() per distinct name plus traceable growth allocations for every per-group vector, all taking the global GC allocation lock - collect the (name, value) pairs into a plain std::vector and group them by stable-sorting on the name. The vector doesn't need to be visible to the GC since the values it points to are kept alive by the list in args[1] for the duration of the call. The shared machinery (the NameValue type, sorting/counting the names, and iterating over the per-name runs) is factored out into helpers used by both primops. This removes the last user of ValueVectorMap, so drop that typedef. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
builtins.groupBy: Drop ValueVectorMap
Evaluating an empty attrset literal wrote its position into
Bindings::emptyBindings, the shared static object that
EvalMemory::allocBindings() returns for zero-capacity bindings. Under
parallel evaluation this is a data race (the position of every '{ }'
was whichever one was evaluated last), and 'perf c2c' on an Intel
i7-1260P showed it also causes false sharing: emptyBindings happens
to share a cache line with Counter::enabled, which is read by every
thread in allocValue()/callFunction()/maybeThunk(), so each write
invalidated that line across all cores.
Skip the write for the shared empty bindings; '{ }' now has a
deterministic (undefined) position instead of a racy one, and the
cache line stays clean. Verified with perf c2c that the line no
longer appears among contended cache lines.
Assisted-by: Claude Fable 5 <noreply@anthropic.com>
ExprAttrs::eval: Don't write to the shared empty Bindings
Introduce LocalStore::addTempRoots, which adds all temproots in a batch for LocalStore. Make this available from the base class Store by making a virtual method Store::addTempRoots, overriding it in LocalStore and RemoteStore (it is just a loop for each path because the benefit of having this be batched is unclear), and making the single-path Store::addTempRoot a wrapper around that.
Since 734a205, copyPaths() calls addTempRoot() on the remote store for every path in the closure. However, this is extremely show for large closures, especially over high-latency SSH connections. So now, we use a new daemon operation AddTempRoots to do all paths in a single call. Fixes #533. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
Use the new batched addTempRoots for both convenience and better performance for LocalStore.
Use the new batched addTempRoots for both convenience and better performance for LocalStore.
Fix very slow remote temp root registration in copyPaths()
This is intended to allow a store to query multiple paths in a single call. Taken from #523. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
Add a batched QueryPathInfos operation to the worker protocol, gated behind a new "queryPathInfos" protocol feature, and use it to implement an efficient RemoteStore::queryPathInfos() override. Previously, querying path infos from a remote store required a network round-trip per store path, which is extremely slow over high-latency links (e.g. in computeFSClosure()). Now the entire set of paths is sent in a single operation. The response contains the infos for the valid paths; paths not reported are invalid. The client answers what it can from the in-memory path info cache first, updates the cache with the results (including negative entries), and falls back to the per-path base implementation for daemons that don't support the new operation. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
Make computeFSClosure() much faster over SSH
Tagging release 2.34.8
Sync with upstream 2.34.8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Not intended to be merged directly. This PR is a convenience to show the diff between upstream Nix and Determinate Nix (the
mainbranch).Continuation of #4.